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Abstract. This paper reports on a study examining the effects of visualizing online 
writing activities on student behaviors and learning outcomes. A design-based research 
approach was adopted to develop and integrate theory and practice in natural educational 
settings (Anderson & Shattuck, 2012). The study was conducted in an 18-week, 
semester-long undergraduate English for freshman engineering students course. The 
course was in a blended face-to-face/online format, using Moodle, and covering the four 
skills of listening, reading, speaking, and writing. A blog visualization tool, designed as a 
monitoring system visible to the instructor and all students, was implemented throughout 
the semester. Data from 28 students were analyzed, with the students’ written consent. A 
mixed-method design was chosen for triangulation, using three data sources: pre-/post- 
course writing proficiency tests, an online post-course questionnaire, and online writings 
on Moodle. The research found that the self-regulatory mechanism supported a regular 
and high level of work performance of the entire class. The class average scores on writing 
tests showed gradual and steady progress. In addition, the phenomenon was observed in 
which students showed indecision about whether to work more or less relative to the 
performance of other students. Those students eventually deciding to work less showed 
certain regression in writing proficiency at the end, even if they started at a higher 
level than the class average. This situation, called the “to-do-or-not-to-do dilemma,” is 
considered a potential negative effect of the monitoring system. A monitoring system for 
self and others can be a positive factor that improves the regular work performance of a 
learning community, leading to a higher outcome. However, the study suggests the need 
for appropriate counter-measures against possible demoralization in the individual and/ 
or the entire class. 
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1. Introduction 

This paper reports on the final stage of a research study that examined the effects 
of visualizing students’ online behaviors on their learning outcomes. The research 
was conducted in a basic English for Science Purposes (ESP) course for freshman 
engineering students. 

A literature review of major CALL-focused international journals (CALICO, 
Computers and Composition, LL&T, ReCALL, Systems) over the past decade (2001- 
2011) reveals that similar research has not been reported examining the effects of 
visualizing learners’ online writing on their learning outcomes in foreign language 
learning. 

Prior to the current study, two pilot studies found correlations among the frequency 
of using a blog visualization tool, the amount of blog postings, and improvement in 
English writing ability (Miyazoe & Anderson, 2011). However, in a one-semester 
experiment, the observed progress in writing ability was small, and it was not clear if 
this positive effect occurred because of the novelty of implementing new technology. 
In its final stage, this research therefore sought to answer two questions: 1) Does an 
environment in which students can monitor their own and other class members’ work 
performance produce net learning outcomes in terms of writing proficiency? and 2) 
Does this monitoring become a stable learning strategy for the students over a one-year 
longitudinal intervention? 

2. Method 

A total of 28 engineering students in a night program participated in the data 
collection and provided their written consent for analysis and possible publication 
of the results. The course included one year of spring and fall semesters, each 
lasting approximately 18 weeks with holidays and school events. Four-fifths of the 
students participated in both semesters; five students were new to the class in the 
fall. 

The intervention included the use of a blog-viewing tool, regular blog writing, and 
pseudonym use, all developed and viewable on a Moodle course web space. Writing at 
least one blog post per week was recommended, without penalty for not posting. These 
conditions were the same throughout one year. As a natural part of the course design, to 
make it more challenging in the second semester, five optional essays (approximately 
one essay every two weeks) were required in the fall to allow the students to practice 
writing essays in addition to blogs. This assignment was accompanied by a one-time 
guidance session to introduce the academic writing structure before the due date for the 
second essay assignment. 

For triangulation, a mixed-method approach of using both quantitative and 
qualitative approaches was chosen (Creswell & Plano Clark, 2007). Three data sources 
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were consulted: 1) pre- and post-course writing proficiency tests, 2) an online survey, 
and 3) blog and forum essay postings that were stored on Moodle. Of the participants, 
27 students completed the optional online survey. 

The pre- and post-course English writing tests simulated the opinion essay writing 
section in the Test of English for International Communication (TOEIC). The tests 
were prepared and administered in a plain text format on Moodle with a 30-minute 
time limit. Different essay topics that used a compare/contrast structure with accessible 
topics were selectively chosen for pre-/post-test comparison. The digital TOEIC essays 
were copied onto paper scoring sheets, assigned randomized identification numbers, 
and evaluated by two experienced native English higher education teachers; for 
comparability of the results, the same raters were used for spring and fall. 

To increase validity, the raters worked independently without being informed of the 
existence of the co-rater. The order of the essay topics (one was pre-test, and the other 
was post-test) was also not known by the raters. The same scoring rubrics used for 
scoring opinion writing in the Test of English as a Foreign Language (TOEFL) offered 
by the ETS (Trew, 2010, p. 157) were used. The rubrics provided scores ranging from 
0 to 5 points, and the raters were asked to use 0.5-point intervals for accuracy of the 
results and interpretations. 

A post-course online survey was administered in an anonymous format to ensure 
a high response rate. The survey contained a question regarding the frequency of use 
of the viewing tool to verify the possible correlation between tool usage and students’ 
online behaviors. 

The blogs and essay assignment posts on Moodle were copied from the system and 
used for quantitative analysis from various perspectives. 

3. Results 

Because of space limitations, only the critical findings are selectively reported. Overall, 
the students’ participation in online writing was constant throughout the year. The 
average number of blog posts per student was 11.5 (N= 28, SD = 6.95) in the fall and 
11.03 (N= 31, SD = 4.24) in the spring. 

Figure 1 summarizes the total number of blog posts per week and essays per 
each assignment during the semester to depict the students’ overall participatory 
behaviors. Blog and essay writing post frequencies are juxtaposed to highlight the 
possible interrelationships in terms of workload. The seven squares represent the 
total number of essays posted on each day of the week. The two squares in Week 
2 and Week 18 correspond to the two TOEIC-type opinion essay tests, and the five 
squares in between represent the five essay assignments. The decline in Week 16 
corresponded to winter break. In this research, the negative correlation between blog 
posts and essay posts is considered to reflect the students’ strategy for balancing their 
workload between the two assignments. 
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Figure 1. Total number of posts per week during the semester* 



Figure 2 summarizes the average scores of the students who took both the pre and post 
TOEIC-type essay tests in both spring and fall semesters. The slight decrease from the 
spring post-test to the fall pre-test (2.05 to 2.03) can be explained by the inclusion of 
five new students in the fall as well as the two-month summer break between semesters. 
It is noteworthy that from the spring pre-test to the fall post-test, the average score for 
the entire class improved from 1.84 to 2.25 (an increase of 0.41 or 22.3%). Therefore, 
in this research, the writing proficiency of the students made continuous improvement 
over the course of the one-year experiment. 

Figure 2. Change in English writing proficiency over the course of one year 



Figure 3 summarizes the changes in scores on the essay tests and the length of the 
Essay 2 assignment immediately after the essay structure guidance session (on the left) 


* T: Test, E: Essay assignment 
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and the total length of blog and essay assignment posts in word count and the length of 
Essay 2 (on the right). All of the essay lengths were analyzed, and the length of Essay 
2 proved particularly worthy of attention. The data of 25 students who completed both 
tests in the fall were included to find factors that produced the higher learning outcomes. 
Word count was used because in the tradition of research on writing, that statistic is 
considered an indicator of progress in writing ability (Wolfe-Quintero, Inagaki, & Kim, 
1998). A high correlation was found between the change in test scores and the length of 
Essay 2 (Figure 2: left side, r = .675, p < .01). Moreover, a strong correlation between 
the post-test length and the total length of the blogs and opinion essays that each student 
produced (Figure 2: right side, r = .672, p < .01) is noted. This may suggest that among 
the five essay assignments, the length of Essay 2 was the most accurate predictor of the 
progress that the students would demonstrate toward the end of the course. 


Figure 3. Post length and change in writing proficiency. 



Finally, further analysis focusing on those students who made progress (improved 
group) and those who regressed (regressed group) over one year reveals that viewing 
the lower performance (in writing ability) of other members may have discouraged 
those with initial high scores from making efforts to improve. At the end of one year, 
the initial high scorers were overtaken by the initial low scorers. 

4. Discussion 

This research suggests that monitoring both one’s own and others’ work via the viewing 
tool appeals to the self-regulatory mechanism of a learning community supporting 
the regular and high levels of work performance of the entire class. The students in 
this study demonstrated adequate progress in writing ability over a one-year course 
period. However, the study also observed evidence of two diverging directions in class 
members who were motivated to work more and those who were motivated to work 
less, who eventually regressed in the targeted skill. The authors call this struggle of 
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indecision whether to do more or less the “to-do-or-not-to-do dilemma,” describing a 
situation in which a student hesitates about the degree of commitment to the workload 
of a learning community. 

A monitoring system of work performance for self and others can be a positive 
factor that improves the regular work performance of a learning community, leading to 
a better outcome. However, the study suggests that this monitoring system should be 
accompanied by appropriate countermeasures against possible demoralization in the 
individual and/or the entire class. 
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